1. Field of the Invention
The present disclosure relates generally to modular packet network devices such as switches and routers having separate data and control planes.
2. Description of Related Art
Packet network devices direct data packets traveling across a network between data sources and destinations. Packet network devices can perform “routing” or “switching” depending on the header information and networking techniques used to direct the data packets. For purposes of the following disclosure, the differences are not of particular interest, and therefore the described packet network devices, no matter how the devices determine egress ports for data packet traffic, will be referred to as “switches” for simplicity.
FIG. 1 shows a high-level block diagram for an exemplary modular packet switch 100. The switch comprises a chassis (not shown) with an electrical backplane 110, to which circuit cards mate in order to communicate with other cards in the chassis. In FIG. 1, the switch accommodates n line cards, LC1-LCn, m switch fabric cards, SFC1-SFCm, and two route processing managers RPM0, RPM1.
Each line card LCx (where x represents any of the line card slot positions 1 to n) receives ingress data traffic from and transmits egress data traffic to peer devices through its external ports to a port interface, Ph. Port interface Ph transmits and receives the physical layer signaling required by the external ports, performs framing, and communicates packet streams for each external port with a packet processor PPx. The ports can be configured for different electrical or optical media via the use of different line card types, different port interface modules, and/or different pluggable optics modules.
For most ingress packet traffic on each line card LCx, a line card packet processor PPx processes the packet, determines one or more switch egress ports for the packet, and queues the packet for transmission through the switch fabric when possible. For most egress packet traffic on each line card LCx, the line card queues the packets arriving from the switch fabric, and selects packets from the queues and serves them fairly to the egress ports. To accomplish these functions, PPx interfaces with a scheduler interface SCHx, a content-addressable memory CAMx, and a line card processor LCPx. PPx notifies scheduler interface SCHx as to the destination line cards for which it has packets queued, and receives switch fabric schedule information back from SCHx. PPx uses the scheduling information to place packets from one or more queues on the switch fabric backplane connections during each scheduling epoch. The switch fabric connections are implemented using unidirectional differential serial bus pairs, with each line card packet processor having at least one ingress pair for each switch fabric card and one egress pair for each switch fabric card. An intermediate serializer/deserializer (“serdes,” not shown) may be used between PPx and the backplane differential serial pairs.
Memory CAMx stores lookup tables that PPx accesses to determine what operations to perform on each packet, as well as the next hop destination for each packet. PPx generally constructs one or more lookup keys from the headers and receive port information, etc., and performs one or more lookup operations on CAMx. CAMx returns pointers into instruction memory and next hop memory (not shown), as well as any parameters needed to process the packets.
Line card processor LCPx is a general purpose processor that handles control plane operations for the line card. Control plane operations include programming CAMx and lookup memory according to instructions from the master RPM, programming registers on PPx that tailor the line card behavior, receiving control plane packets (packets addressed to switch 100, e.g., for various routing/switching protocols) from PPx, and transmitting control plane packets (packets generated by switch 100 for communication to a peer device) to PPx for forwarding out an external port. LCPx may implement some control plane functionality for some protocols handled by switch 100.
LCPx also connects to the electrical backplane through a card management FPGA (Field Programmable Gate Array) FPGAx and an Ethernet switch ESWx. The card management FPGA connects through serial management bus interfaces on electrical backplane 110 to master card management FPGAs on the RPMs. The RPMs use the serial management bus to boot line cards, monitor the health of the line card and its environmental parameters, manage power for the line card and its components, and perform basic hardware configuration for the line card. Various registers on FPGAx are readable by line card processor LCPx, allowing LCPx to perform some of its configuration tasks using values stored in the registers by the RPMs.
Ethernet switch ESWx connects to two backplane Ethernet point-to-point links, one linking ESWx to each RPM. LCPx transmits control plane traffic, as well as line card statistics and monitored data plane traffic, to the master RPM using packets transmitted to ESWx. Likewise, the master RPM transmits FIB (Forwarding Information Base) updates and boot images to LCPx for consumption, and control plane packets destined for external peers to LCPx for forwarding to PP1.
Switch fabric card SFC1 is exemplary of the switch fabric cards, which are generally identical in a system. Switch fabric card SFC1 comprises a switch fabric device, SF1, a switch fabric scheduler interface SFSCH1, and a card management FPGA1. Although FPGA1 can be the same type of hardware device as FPGA1 on line card LC1 and have the same RPM interface programming, its other programming is generally specific to the requirements of a switch fabric card. The scheduler interface SFSCH1 does not provide schedule information to each RPM, it merely receives the epoch schedules used to program the switch fabric for each epoch. The switch fabric cards work in parallel according to a common epoch schedule. Each epoch, every switch fabric card has a switch fabric SFy that is programmed the same way through the switch fabric scheduling interface SFSCHy.
The switch fabric SF1 provides serdes interfaces for each line card and a parallel crossbar switch that can switch any of the inputs to any number of the outputs. In one embodiment, the line cards slice up each packet sent to the switch fabric into lanes, sending one byte of the packet to SFC1, the next byte to SFC2, etc., in round-robin fashion on each lane. A receiving line card aligns the lanes incoming from the switch fabric cards and reassembles the packet data. This allows the switch to tolerate malfunctioning switch fabric cards by simply not sending data on a malfunctioning lane.
The route processing managers (RPMs) control all aspects of the overall operation of the chassis. RPM1 and RPM2 are identical, and the switch can run with only one RPM. When two RPMs are present, one is elected as the master, and the other remains on standby. The standby RPM monitors the health of the master, and takes over as master should the first fail. Each RPM RPMx comprises three processors: a control processor CPx, which controls the overall operation of the switch; and two route processors RPx.0, RPx.1, which run different routing/switching protocols, communicate with external peers, and program the line cards to perform correct routing and switching.
Each RPM RPMx also has three bus interfaces to the electrical backplane. A master Ethernet switch MESWx connects through backplane Ethernet links to each line card Ethernet switch, ESWy, and to the control processor and each route processor. A master scheduler MSCHx connects through the backplane scheduling bus to each line card and switch fabric card—the master scheduler determines from the line card requests a switch fabric schedule for each epoch, and communicates this schedule to the line cards and switch fabric cards. A master FPGA MFPGAx connects through backplane serial management bus connections to each other card in the chassis. The master FPGA monitors the health of each other card through heartbeat message exchanges and environmental data collections, and provides an interface for CPx to control the hardware on each card.
As alluded to above, communications between the chassis modules uses four separate bus systems on electrical backplane 110. A packet data bus system connects the line cards and the switch fabric cards, allowing high-speed transfer of data traffic through the switch. Due to the high data rates of this bus and the fact that separate trace groups connect each of n line card slots to each of m switch fabric card slots, this is the largest bus system on the electrical backplane 110. A control plane packet data bus system allows the master routing/switching protocols running on the RPMs to communicate with each other, with the line cards, and with peer devices through the line card external ports. This packet data bus system may be, e.g., a point-to-point Ethernet link between each line card and each RPM card. A scheduling bus system allows the master RPM to coordinate and synchronize the internal transmission of packet data between the line cards and the switch fabric cards. This bus system is separated to preserve critical timing for the exchange of scheduling information between each line card and switch fabric card and each RPM. And a chassis management bus system allows the RPMs to monitor, configure, and manage the hardware on each line and switch fabric card. This bus system may use industry standard relatively low-speed serial management bus protocols, such as System Management Bus (SMBus), across point-to-point links between each line card or switch fabric card FPGA and each master FPGA.
FIG. 2 illustrates one embodiment of electrical backplane 110 in perspective view. Backplane 110 distributes power and signaling between the various card described above, using connectors aligned with card slots. A top row of card slots provides two central RPM slots and fourteen line card slots, seven outboard of the RPM slots to the left of the RPM slots and seven outboard of the RPM slots to the right of the RPM slots. A bottom row of card slots provides nine switch fabric card slots. Outboard of the switch fabric card slots on either side, power supply connection regions 120A and 120B allow connection of A and B redundant power supplies to backplane 110 in order to distribute power to the cards.
Each card slot provides power connectors and signaling/digital ground connectors. For instance, two power connectors LCPC1A and LCPC1B provide A and B power from the backplane to the first line card slot, and a signaling connector LCSC1 provides signaling connections for each of the four described bus systems to the first line card slot. Although not all connectors have been labeled, those that are labeled include: A power connectors, RPPC0A and RPPC1A, for RPM0 and RPM1 card slots, respectively; signaling connectors, RPSC0 and RPSC1, for RPM0 and RPM1 card slots, respectively; the signaling connector LCSC14 for the last line card slot; the signaling connectors SFSC1 and SFSC9 for the first and last switch fabric card slots, respectively; and a representative power connector SFPC5B for B power delivery to the fifth switch fabric card slot. The other, unlabeled connectors provide similar functionality to cards inserted in the corresponding slots.
FIG. 3 shows the same perspective view of the backplane 110, this time with two line cards (LC11 and LC13) and three switch fabric cards (SF5, SF6, SF7) connected to the backplane. The switch chassis and chassis card carriers have been omitted for clarity—in an actual system, the chassis and card carriers serve to partition each card from its neighbors to control electromagnetic interference (EMI), while allowing a common cooling solution. As shown, line card LC13 contains a logic board LB13 and a port interface module PIM13. Port interface module PIM13 couples signals from eight external port modules (exemplary module P13-1 is labeled) to and from the logic on logic board LB13. Different port interface module types can be used to couple a desired port mix to the line cards.
FIG. 4 shows a side view of backplane 110, with the front sides of the attached line card LC13 and attached line card SF7 visible, showing further details of the physical assembly. Within backplane 110, four thick central power planes distribute A and B power from the power supply connectors (120A and 120B, FIG. 2) to the power connectors LCPC13A, LCPC13B, SFPC7A, and SFPC7B. These four central power planes lie between several low-speed signal layers, serving for instance scheduling bus traces, serial management bus traces, and backplane Ethernet traces. Outboard of the low-speed signal layers, high-speed signal layers serve the data plane switch fabric-to-line card differential pairs, and other signals when possible. Between adjacent high-speed signal layers, between the adjacent high-speed and low-speed signal layers, and between adjacent power delivery and low-speed signal layers, digital ground layer provide noise isolation and a single-ended impedance reference. The digital ground layers and the traces on the signal layers connect to designated pins, e.g., on signal connectors LCSC13 and LCSC7, to provide signaling functionality between the various system cards.